Neural Network models, which evolved from efforts to understand neurophysiological and psychological phenomena, are now being considered for performing various artificial intelligence tasks. However, further advance in this field is limited due to the absence of reliable and efficient hardware realizations of neural network models.
A typical neural network model consists of many simple processing units called the neurons, each connected to many or all the others. The state of the network is the collective state of the neurons, and the data stored in it is distributed as the strength of the connections between the neurons called synapses. Consequently, realizing neural network models using electronics poses severe problems:
a) The synapses require a very complex wiring scheme virtually impossible to achieve with integrated circuit techniques;
b) "Teaching" the network to modify the synapses is a slow process in an electronic system which must be addressed serially through its regular input/output ports.
The neurons can be binary: e.g., V={0,1}, or they can be analog: e.g., V=[-a,a] where "a" is some real number. As just noted above, the neurons are interconnected by synapses, the strength of which is given by the synaptic interaction matrix W, where W.sub.ij designates the strength of the connection from j'th neuron to the i'th neuron. Each neuron is updated according to the total current that flows into it, namely I.sub.i for the i'th neuron where EQU I.sub.i =.SIGMA.W.sub.ij * V.sub.j
By using some decision process, such as designated by the equation EQU V.sub.i =.phi.(I.sub.i, b.sub.i)
where b.sub.i is a constant parameter of the i'th neuron, the next state V of the i'th neuron is determined.
One possible application of neural network models is as content addressable (associative) memories. In that application, a set of p N-dimensional vector s, V.sup.(s) s=1 . . . p, are stored in the network by modifying the synapses using some "learning rule". For example, the learning rule used by Hopfield to store V.sup.(s), s=1 . . . p, N-dimensional vectors in a binary N-dimensional network is ##EQU1##
It was shown by Hopfield that each of the stored vectors V.sup.(s) will be a stable state of the network. Moreover, it is expected that if the system is not in one of the stable states (V.sup.(0) .noteq.V.sup.(s), s=1 . . . p), it will be attracted to the stable state which is the closest to its initial state. The W.sub.ij 's are not necessarily computed in advance and loaded into the network prior to its operation. One of the interesting features of neural network models is the ability to perform independent learning. Here the W.sub.ij 's are modified according to the reaction of the network to some teaching pattern, as done for example when the "back error propagation" learning method is applied.
The neural network models can be updated either synchronously or asynchronously. In the synchronous case, the neural network model is updated in cycles. For each cycle, the state of each neuron is set according to the state of the neural network model in the previous cycle. In the asynchronous case, each neuron is updated according to the state of the neural network model at the updating moment.
It is well-known in the neural network or intelligent computer arts that a synapse is one name given to a matrix element of a matrix in which an input vector (representing the input neurons) multiplies the matrix elements to produce an output vector (representing the output neurons). In many applications such matrices or plurality of synapses are implemented in the form of integrated circuits on a single chip. It is often critical to minimize the size of the surface area that is required on the chip to implement each synapse or matrix element. The size of each matrix element is generally directly proportional to the number of transistors required to implement it and also to the capabilities within each such matrix element. By way of example, it generally requires more transistors within each synapse or matrix element to provide a learning capability wherein the weight of each matrix element may be altered in what is commonly known in the art as an outer-product learning implementation. In such an implementation, the weight is changed in proportion to the product of the input and output weight change vectors corresponding to that particular matrix element. Learning is an important characteristic of neural network applications because it tends to simulate what is believed to be actual neuron operation wherein the effect of each operation changes or modifies the communication characteristic of the neuron by altering the synapse behavior. The learning described here is `Hebbian` Learning where the weight change is proportional to the product of the input and output states. EQU .DELTA.W.sub.ij =V.sub.j V.sub.i
Other Learning rules (back error propagation) are more complex in that the outer product vectors can be different from the input and output vectors. EQU .DELTA.W.sub.ij =.beta..sub.j .delta..sub.i
One typical implementation of a synapse or matrix element used in neural network applications in the form of an integrated circuit chip, comprises an Intel model 80170NW wherein each synapse requires six transistors occupying a total solid state surface area of approximately 50 by 50 micrometers, or approximately 2500 micrometers squared. Unfortunately, this particular typical prior art synapse implementation does not provide an on chip learning capability.
There is therefore a need to provide an implementation of a synapse for neural network applications, wherein not only is the size reduced in order to permit an increased number of synapses or matrix elements on one chip, but wherein the number of transistors is reduced per matrix element and wherein a learning capability is provided in-situ.